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-Abstract- 

Aiming to offer a framework for blended learning to the teaching of proof theory, the present pa¬ 
per describes an interactive tutorial, called TryLogic, teaching how to solve logical conjectures 
either by proofs or refutations. The paper also describes the integration of our infrastructure 
with the Virtual Learning Environment Moodle through the IMS Learning Tools Interoperability 
specification, and evaluates the tool we have developed. 
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[Y] Introduction 


“(■..) Knowledge never hurts — what hurts is helplessness, the futility of 
hanging your head against a brick wall without finding either proof or dis¬ 
proof. I have often spent weeks trying to prove a false statement — and when 
I learned that it is false, I felt victorious. Progress was made, knowledge was 
acquired, one more step toward the truth was taken.” 

— Paul R. Halmos 

I Want to be a Mathematician: An Automathography (1985) p. 91. 


Logic permeates computing and provides essential tools for dealing with data structures. 
The curricula of discrete mathematics and logic courses emphasize a kind of verificationist 
approach through the teaching of techniques that focus on proving, instead of approaches 
that encourage also disproving, or refuting by way of counter-examples. The main afteraffect 
of such approach is a common misunderstanding by the students of the boundaries between 
a deductive framework and a semantic refutation. 

Logic is essential, in particular, for verifying correctness of code. In checking that an 
iteration behaves as expected, for instance, one has to: (i) carefully choose an appropriate 
boolean condition to test, (ii) check whether the given initial conditions of an iteration imply 
the required postcondition, and usually also (hi) prove that the execution of the corresponding 
code terminates. In case the implication mentioned in step (ii) is refuted, then the iteration 
code does not implement the desired specifications for the postcondition. Conversely, if it 
is shown that the preconditions do imply the postcondition, then the iteration does satisfy 
the desired specification. The heuristic used in the analysis of this sort of situation involves 
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basically the challenge of finding out whether certain conjectures can be proved or refuted. 
The fact that the two latter tasks, proving and refuting, are complementary is a very practical 
application of the soundness and completeness metatheorems. 

The aim of the present paper is to present a tool for teaching the use of logical reasoning 
to verify conjectures for which it has not previously been determined whether they are 
provable or refutable. One of the main goals of our tool is to teach how to construct a fully 
justified counter-example to witness the falsity of a given conjecture. This tool implements 
some of the teaching principles discussed in [S] . 

In the current state-of-the-art, an approach not unlike ours is used in Bornat’s |2], 
where Logic (formal deductive proof, formal semantic disproof and program specification) is 
presented with the help of the proof assistant JVp3. Many other existing tools also combine 
the teaching of Proof Theory with Formal Semantics, e.g. AproS Project, Panda, Tarski’s 
World, Fitch and Boole. On top of those methodologies and tools, our contribution is to 
track learning beyond the mere use of proof strategies. Continuing the work presented by 
Terrematte et al. [7], here we present an interactive tutorial to guide the student through 
the process of learning by trial and error: the TryLogicQ 


“(...) mathematics is not a deductive science. When you try to prove a the¬ 
orem, you do not list the hypotheses, and then start to reason. What you do 
is trial and error, experimentation, guesswork. You want to find out what 
the facts are, and what you do is in that respect similar to what a laboratory 
technician does, but it is different in its degree of precision and information.” 
— Paul R. Halmos 

I Want to be a Mathematician: An Automathography (1985) p. 321. 

Logic courses represent a pedagogical challenge and the recorded number of failures and 
discontinuities in them is often high. On that track, the main goal of TryLogic is to 
diminish the cognitive overload through a step-by-step tutorial, presenting different topics of 
logic related to the process of proving or refuting logical conjectures. Our tool TryLogic 
aims to: 

H present a set of lessons in Propositional Logic that exemplify the task of proving in 
natural deduction (theory Np) and refuting in a formal semantics (theory Senip) through 
Coq; 

H organize Logic in an interactive way and provide self-evaluation tasks to students; 

H provide the teacher with a follow-up activity report on the lessons completed by the 
student at ProofWeb, and provide a multi-language infrastructure for human-machine 
interaction. 

The framework is implemented by combining the following tools: 

H ProofWel:0 an open source software for teaching Natural Deduction which provides 
interaction between some proof assistants (Coc^Isabelle, Lego) and a Web interface [3]. 
H Conjectures Generatoij^ a tool for task generation of a set of conjectural arguments 
(i.e. a set of premises with a formula that may follow or not from these premises). This 
tool was developed by our group. 


2 The TryLogic tutorial for proving and refuting 


^ Available at http://lolita.dimap.ufrn.br/trylogic 
^ Available at http: //prover .cs.ru. nl/ 

^ Available at https: //coq. inria. f r/ 

Available at http://lolita.dimap.ufrn.br/logicamente-ge/ and it is open source code is available 
at http://github.com/terrematte/logic-propgenerator 





P. Terrematte and J. Marcos 


235 


H TryOCamlj^ an infrastructure consisting of an interactive tutorial for teaching and inter¬ 
action with the functional programming language OCaml. 

H Moodle: a well-known Virtual Learning Environment (VLE) that helps in organizing 
contents and educational activities. 

H IMS Basic Learning Tools Interoperabilitj/]^ (IMS LTI): a specification for the im¬ 
plementation and integration of educational tools. 


U Figure 1 Lessons on TryLogic integrated to Moodle 


Logic Applied to Computing 


My homefl LACf 


1‘utorial TryLogic 


You are logged in as 


Tasis on TryLogic v ^ Lesson 5 ^ step l ^ En-Ev 

Rules for Negation I 


In Propositional Logic, we can define negation (-«) by implication and a 
constant representing an absurd (J.). 


The rules of negation (-> I) and {-• E) are similar to the implication rules 
(-.1) and (-* E) 


Let's prove that: " I- a -* a)". 


Reset Initial. 

Require Import ProofWeb. 
Variables A : Prop. 

Theorem exDN5_l : —("A -> A). 
Proof. 


Reset Initial. 

Require Import ProofWeb. 
Variables A : Prop. 

Theorem exDN5_l : —(—A -> A). 

Proof. 

neg_i hi. 

neg_e (—A -> A). 

apply hi. 

[imp i h2. 
neg_e (-A). 
apply h2. 
neg_i h3. 
neg_e (-A), 
apply h2. 


The main connective is a negation, so the strategy is to use the rule (-> 
I), and we obtain a hypothesis hi: 


neg_i hi. 


O so a 5 O 


Click here to execute this code 


Now. we set from which rule is obtained the absurd L. Let’s say that -L 
came from {-• E) on -.(-•-.A -* A) e (->-'A -» A), to use our hi: 


1 subgoal 

hi ; - - A •> A) 

h2 : - ~ A 
h3 : A 


- th2l 


-i(h3) 


-(--A - A) 




The TryLogic was developed as branch of the TryOCaml project, i.e., all lessons and 
interaction follow the same architecture of the latter. We implemented the Senip theory in 
Coq and integrated the whole system with ProofWeb and Moodle. With a goal of centralizing 
the use of our tools, we have used the specification IMS LTI, which is, according to the survey 
[I], one of the most representative alternatives to infrastructure integration between teaching 
platforms. Using this specification, any collaborator who wishes to use TryLogic in any 
VLE can add it as an external tool. This way, it is not necessary to install TryLogic nor is 
it necessary to obtain special access permission for the server in which it is being installed. 


® Available at http: //try. ocamlpro. com/ 

® The IMS LTI was developed in 2006 by IMS Global Learning Consortium and is available at http: 
//www.imsglobal.org/lti/index.html 
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2.1 The Conjectures Generator 

The Conjectures Generator for Propositional Logic was implemented through a formula 
generator with the SAT-solver Limbool^to evaluate propositional formulae. 

The Conjectures Generator creates conjectures in the format of individualized tasks 
for Coq, directly in each student’s area in ProofWeb. The students receive each task in a 
template for them to try and prove (showing that T h a) in the Np theory and another 
one for them to try and refute (showing that T a) in Senip theory. Of course, soundness 
and completeness connecting the two theories guarantee that only one of these tasks can be 
fulfilled. 

The Conjectures Generator was implemented with requirements that allow one to 
establish some connections of relevance between the premises and the conclusion, namely: 
that both the conclusion and the conjunction of the premises should be contingent, and 
that each formula of the premises must share at least one atom with the conclusion. Other 
settings are available, e.g. choosing the number of conjectures, number of premises, number 
of distinct atoms, selecting the connectives and a range for the complexity of the formulae; 
also, the user may decide if in the generated conjectures all premises are necessary to prove 
the conclusion, and if the collection of generated conjectures are all provable, all refutable, 
or are evenly divided into provable and refutable, to be randomly assigned to the students. 

Through the available settings, the Conjectures Generator is a useful tool for the 
teacher who wishes to evaluate propositional arguments through truth-tables, tableaux, 
natural deduction, resolution methods and even produce tasks concerning the evaluation of 
arguments. 


3 I Propositional logic for proving and refuting 

“(...) Every genuine test of a theory is an attempt to falsify it, or refute it.” 
— Karl Raimund Popper 

Conjectures and Refutations: The Growth of Scientific Knowledge (1963) p. 36. 


To prove a conjecture T h a in propositional logic with Natural Deduction it is necessary to 
build a derivation tree to witness it. Our students were taught to do this using the rules of a 
theory we call Np, which is essentially the same that is natively implemented on ProofWelj^ 
following the usual style of natural deduction introduced by Gerhard Gentzen in 1935. As 
an illustration, the rules for disjunction are the following: 


_ m 

a 




n 


P 

ay p 


(V/o) 


a 

ay p 


(V/l) 


ay P 7 

t" 


7 


(VS):m,n 


In constrast, in an approach involving formal semantics, we build a refutation tree by 
using the notions of valuation and satisfaction. A valuation v maps formulae to truth-values. 
An argument is refutable (T a) if there is a valuation v that satisfies all formulae in T and 
simultaneously falsifies a (u Ih T and u lb a). To refute conjectures the students need to 
build refutation trees on the Senip theory. The rules of Senip compositionally manipulate 
satisfaction of formulae by a valuation v, and they are the following: 


^ Available at http://fmv.jku.at/limboole/ 

® Check the ProofWeb’s manual at https://prover.cs.ru.nl/maii.pdf 






P. Terrematte and J. Marcos 


237 


V 




a 


(- T) 


V II —la 
V \\- a V \\- p 


v\\- a f\ (5 
V \\- a 


V II- 
vV^- 

(A T) 
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(-L) 
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T Ih a T Ih /3 
T Ih a o /3 


T Ih a V /3 
Tlh/g 
T Ih a 


P 


(<->■ Tl) 

(AA Fl) 


(V T2) 


(-1 T2) 


T Ih a 


V Ih a A P 
vllha v\>hp 
vlh ay p 
T Ih a vVhp 

P 


(A F2) 


T Ih a 
Tlh/3 


(V F) 


F) 


T Ih a o /3 
T Ih a T Ih /3 
T Ih a o /3 


(aa Tl) 

(aa F2) 


With these rules, the Senip theory allows us to show that a given sentence is not a semantic 
consequence of a given set of premises. Here is a full example of a refutation tree: 


U Ih p 


Ih r 


(- 


u Ih q V (r p) 


.T2) 

(VT2) 


u Ih p 
u Ih ^p 


(-F) 


Ih r 


ulh ■ 


; Ih ^p V ^r 


V Ih (^p V ^r) q 


(^F) 
- (VF) 
(^Tl) 


u Ih p T Ih q 
u Ih p o q 
V II—^(p O q) 


(aaF) 

(^T) 


Ih r 


ulh 


(^F) 


' Ih A p 


ulh^(peAq) (^rAp) 


(afi) 

(^F) 


q V (r ^ p), (^p V ^r) -> q h ^(p O q) -> (^r A p) 




In a bottom-up reading each connective in Senip has rules that provide a sufficient condition 
for a valuation v to satisfy (or falsify) a given sentence. On the other hand, in a top-down 
reading, the application of the rules represent a semantic inference. In the branches of the 
refutation tree one finds statements in the form u Ih a or u Ih a. In the leaves, one finds 
statements such as u Ih p or u Ih p, where p is an atomic formula. A refutation tree represents 
thus a fully justified counter-model to a given conjecture. Note that the rules are analytical, 

i.e. the premises of each rule contains statements over subformulas of the formula in the 
conclusion of the rule. This ultimately means that the leaves of an exhausted tree are always 
over atomic formulae. 

The general backward strategy for building a refutation tree follows these steps: 

1. Assume that the conjecture is refutable, i.e. that there is a valuation v that satisfies its 
premises and falsifies its purported conclusion. 

2. Exhaustively explore and justify step-by-step with the Senip theory the consequences of 
the initial assumption that the conjecture is refutable. The exhaustive exploration means 
it may be necessary to backtrack to try other possible rules. 

3. Check if the valuation consistently satisfies or falsifies each propositional atom involved, 
i.e. it cannot be that a valuation v satisfies some atom p (v Ih p) and falsifies the same 
atom p (u Ih p) in the same refutation tree. 

Note that the above strategy only applies to refuting contingent or contradictory formulae, 
and to exhibiting witnesses to invalid arguments. If one does not manage to build a refutation 
tree after an exhaustive exploration of the possibilities, this means that there does not exist 
a valuation v such u Ih T and v Ih a. Therefore, by the relation of semantic consequence 
we know that T 1= a. Thus, from the completeness theorem (T N a T h a), one knows 
that the conjecture can be proved, i.e. that it is possible to build a derivation tree in the JVp 
theory. 
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3.1 Some logical and pedagogical remarks 

Each connective rule of the Senip theory is implemented by tacticals in Coq and we extended 
the Proof Web system to display the corresponding refutation trees, as illustrated in Fig.|^ 

M Figure 2 Script for refutation on ProofWeb 


Reset Initial. 

Require Import Semantics. 
Parameter A B C D : Prop. 
Hypothesis fl : (v I|-/- A). 
Hypothesis f2 : (v I|-/- B). 
Theorem sem_ex2 : 

(v II-/- ((-A->B) /\ (-A/\-B))). 

Proof. 

conjFl. 

impF. 

negT. 

apply fl. 

apply f2. 


v| I-/- A 
v| I -- (-A) 


apply[fl] 


-T 


v| I-/- B 


v| I -/- (-A - B) 


apply[f2] 

AFi 


v|I-/- ((-A - B) A -A A -B) 


aj' 


Script of refuting tacticals 


b) Resulting tree on TRYLOGiC/ProofWeb 


One of the pedagogical advantages of ProofWeb is its coherence with the modality effect of 
Cognitive Load Theory [HI p.l29]. The idea is that two well connected sources of information 
reinforce the organizing process and facilitate the transfer of information to the long-term 
memory information store. The multiple representation is applied in Fig. where we can 
observe that without the proof script being inserted on the left-hand side (a) it would 
not be possible to check the object on the right-hand side (b) for the tactical sequence 
represents a justification for the refutation produced. The refutations are not static (b), but 
in fact, correspond to the dynamic linear process of their construction on side (a). Thus, 
the visualization of (b) has didactic value as it is also useful to the communicability of the 
refutation structure in (a). 

The heuristic procedure for refutation presented here might be replaced by other deductive 
formalisms in Propositional Logic, such as the sequent calculus, resolution, tableaux or 
even truth-tables. However, we avoid the truth-table method for its fixed exponential 
computational cost (2", where n is the number of distinct atoms in the conjecture) and 
its purely algorithmic character, which we judge not to have optimal pedagogical value. 
Tableaux, on the other hand, are often very efficient in both tasks of proving and refuting. 
However, they also make the procedure fully automatic. While this might be a desirable 
property from a computational viewpoint, from the didactic perspective we claim that 
tableaux create a conceptually undesirable overlap between deductive formalism and formal 
semantics. As a consequence of the exclusive use of tableaux, students are often led to build 
no appreciation at all for the distinction between Proof Theory and Formal Semantics. To 
clarify the meaning of our semantic heuristics, check the comparison between the refutation 
tree and the tableau method in Fig. Note in particular that in using Senip the students 
are forced all the time to take decisions about which tableau branch they should want to 
explore. It is known that in the worst-case scenarios tableaux might be much more costly 
than truth-tables [21 p.62]. A tableau is exhausted only when all the branches are fully 
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M Figure 3 Refutation tree versus Tableaux Method 



explored, and this may depend on the ratio between the complexity of the formulae and the 
atoms that occur in them. Therefore, if a formula has higher complexity than the number of 
distinct propositional atoms, then the tableau analysis may be longer than the number of 
rows in the truth table. In contrast, our heuristic procedure is not fully automatic and the 
wise choice of which path to follow may introduce exponential speed-up. Ultimately, the use 
of Senip simply requires the production of a sequence of formulae corresponding to an open 
branch of a tableau tree. 

Our goal is to improve the logical intuition of students. Therefore, students are told that 
in cases where the semantic heuristics do not allow for a refutation, they should look for a 
derivation tree in Np. On the other hand, when they are having trouble in proving, they 
might well try to refute the selected conjecture. 

3.2 Analysis of experiments with TryLogic 

We performed several experiments in the years of 2012, 2013 and 2014 to evaluate TryLogic 
in blended learning use. The task of proving or refuting was given to the average of 15 
students per semester of Computer Science in the upper undergraduate course of Logic 
Applied to Computing at the Federal University of Rio Grande do Norte (DIMAp/UFRN), 
of which an average of 58% have actively used the system. Through face-to-face classes 
we taught only using theoretical fundamentals, and our biggest challenge was teaching the 
computer-assisted task of proving or refuting exclusively through TryLogic. The main task 
given to students was to prove or refute six to eight conjectures randomly assigned. These 
consisted in two conjectures per each of three or four levels of difficulty. For instance at first 
level (easiest), the conjectures have 3 distinct propositional atoms, with 3 premises and a 
complexity between 2 and 4 connectives per formula. The fourth and hardest level has 6 
distinct propositional atoms, with 4 premises and a complexity of 4 to 6 of connectives. The 
learning goals are to practice formal proof and refutation heuristics, as well as to advance the 
understanding of soundness and completeness metatheorems. At the end of each experiment, 
students answered a questionnaire about their profile, their use of the available tools, their 
difficulties in solving the tasks and their theoretical understanding of the tasks. 

Some general conclusions about the experiments are: 

H TryLogic provides the understanding of the deductive process in Coq to students who 
had brief theoretical contact with the content of Natural Deduction. 

H The students consistently solved more refutable conjectures than provable ones, even if 
they have received in average an equal number of each kind of conjecture. For instance 
in Spring 2013, out of 60 solved conjectures, the students presented 43 refutations. It is 
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possible to draw at least the following two interpretations for this phenomenon: that the 
search for a refutation tree is easier than the production of a natural deduction proof, or 
that the lessons for proving in Natural Deduction on TryLogic need to have an improved 
teaching strategy. The first interpretation is coherent with our learning goals, we aim to 
show that refutation is natural and necessary in Logic. As for the second interpretation, 
we feel it important to add that some conjectures given as task are really large and 
difficult to prov^and this might explain the smaller number of produced formal proofs. 
B A negative conclusion drawn from the questionnaire was that the practice involved on prove 
or refute, does not necessary imply the theoretical understanding of the metatheorems of 
completeness and soundness. 

H Using the theories Np and Senip, implemented in Coq, the student applies a heuristic for 
proving and refuting through justified and verified steps. This way, with TryLogic, the 
experimental process of ‘trial and error’ is taught in a guided environment. 

Future Works and Final Remarks 

“(■..) we teach mathematics to the engineers, physicists, biologists, psychologists, econom¬ 
ists — and mathematicians — of the future. (...) It is not enough to teach them everything 
that’s known—they must know also how to find out what has not yet been found.” 

— Paul R. Halmos 

/ Want to be a Mathematician: An Automathography (1985) p. 322. 

This paper presents an infrastructure of integrated tools for the teaching of Logic with focus 
on: (i) an organized step-by-step presentation of the content of Natural Deduction and 
Propositional Semantics in a sequential and interactive way; (ii) providing the student with 
interactive self-evaluation tasks; (iii) the interaction with the Conjectures Generator and 
TryLogic with Moodle through IMS LTI. It is worth noting that, since the TryLogic 
is based on Proof Web and the lessons are structured on TryOCaml, our infrastructure is 
extensible and customizable^ to build lessons on any other formal theory implemented on 
Coq or Isabelle, e.g. on Modal Logic, Number Theory, Set Theory, or Hoare Logic. Our 
contributions to teaching Logic are part of an initiative that needs to be enhanced. Some 
opportunities for the extension of the project would include: 

H Producing lessons in English, Spanish, French and other languages. 

H Implementing in the Conjectures Generator metrics of difficulty for derivations given 
by the size of normalized proofs and the number of uses of certain rules in the latter 
proofs. 

H Extending the Conjectures Generator and the theory Senip to First Order Logic and 
producing new tasks and lessons of First Order Logic through TryLogic. 
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PIRSES-GA-2012-318986, funded by EU-FP7, and by CNPq / Brazil. The authors also want 
to thank all undergraduate students of Computer Science and Computer Engineering who 
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® For an example, a conjecture generated in the fourth level to be proved in Natural Deduction was this 
one: 

{^(pV((q ^ (u -H- r)) A(s r))),t ^ (^(rV(rV(p AA p)))),-'((-'((p (-.t)) -H- u)) Vp), ((((rAs) Au) 

(s V p)) el u) A s} h (u ^ (u A (t —>■ (s ^ q)))) V (r AA p). 

^"^The project can be forked from: https://github.com/terrematte/trylogic 
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